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About the National Math and Science Initiative 

NMSI is a non-profit organization that was launched in 2007 by top leaders 
in business, education and science to actively promote solutions to the math and 
science crisis in America. NMSI’s mission is to replicate programs with proven 
results on national scale in order to have a positive impact on America’s 50 
million student public school system. In its first year, NMSI has: 

+ Awarded grants to six states to replicate the successful Advanced 

Placement Training and Incentive Program, which greatly increases the 
number of students succeeding in more rigorous math and science 
coursework. In fall 2008, enrollments were up more than 70 percent in 
Advanced Placement math, science, and English courses in the states 
that received grants from NMSI. 

+ Awarded grants to 13 universities to replicate the successful UTeach pro- 
gram, which encourages math and science majors to enter the teaching 
profession. As a result, in fall 2008, more than 1,000 students around the 
country were enrolled in UTeach programs, creating a new wave of math 
and science teachers for our country. 

NMSI is marshalling public-private cooperation around the country thanks 
to initial funding from Exxon Mobil Corp., the Bill and Melinda Gates 
Foundation, and the Michael and Susan Dell Foundation, with generous in-kind 
assistance from IBM and Perot Systems. 


Forward 

Far too often in the United States, our approach to educational problems 
has been to start pilot program after pilot program. While that unleashes a lot 
of bright ideas, it does not lead to sustained results. Another way of putting it is, 
we’ve lighted a least a million pilots in this country, but we haven’t ignited the 
central heating system. 

Today’s challenges require a new kind of philanthropy - a much more 
entrepreneurial philanthropy - that produces measurable results. Today’s leaders 
in the private sector want to see hard metrics that show programs are working. 
Today’s investors want to know that programs have sustainability and won’t 
disappear when the calendar changes. 

We believe the National Math and Science Initiative is establishing the kind 
of 21st Century paradigm for public-private partnership that has been needed. 
NMSI is expanding programs with proven, quantifiable records of success and 
marshalling the funding that is essential to incubate that success across the 
country. In effect, NMSI serves as a holding company, providing not just financial 
support, but on-going expertise to ensure that grantees grow strong enough to 
stand on their own footing. 

We believe that “funding to scale” provides the clarity and common sense 
that many philanthropists and business leaders have been seeking. Funding to 
scale helps ensure that their contributions will leave a lasting legacy. This paper is 
designed to explain how that model of philanthropy works. I hope you will agree 
that making good programs more accessible to more people is the approach our 
country needs to move forward. 


Tom Luce 



CEO 

National Math and Science Initiative 
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What Is Scaling Effective Programs? 

Scaling Effective Programs is a category of giving that is quite unique. Philanthropists 
have many different interests that guide their giving, but Scaling Effective Programs offers an 
approach that can produce lasting transformation. 

This guide speaks to funders who: 

+ view their giving as venture capital that stimulates other giving 

+ want to support a program for a limited time, rather than an open-ended commitment 

+ want their giving to have significant and lasting effects that can be measured 

Scaling Effective Programs is based on the principle that a limited period of investment 
giving can be structured successfully to create the greatest chance of having programs spread to 
more locations and be sustained long after the initial investment. This approach allows funders 
to move on to supporting other programs, or support the same program in new locations 
without being indefinitely tied to supporting a local implementation. Scaling Effective Programs 
incorporates planned withdrawal of support and program self sufficiency, which avoids abruptly 
“abandoning” a project. 

Scaling Effective Programs is also based on the belief that public services such as education 

should be funded primarily by public funds, but that public/private 
partnerships can create the impetus to maximize effective programs. 
If done correctly, this type of partnership can point the way to large- 
scale adoption and institutionalization. 

There is a certain science to this type of giving. The purpose 
of this guide is to describe the model for those who desire a lasting 
effect for their investment. The model itself is not complicated, 
but it requires a level of focus and discipline not always found in 
the philanthropic world. As so aptly said by Tom Luce, CEO of the 
National Math and Science Initiative, “You can choose to be one of 
a thousand pilot lights, or you can choose to light the furnace.” 

If you want to spark meaningful change in an arena that is important to you, this guide is 
a must-read. 

The Basic Model 

The National Math and Science Initiative (NMSI) is currently using this funding model. 
NMSI’s mission is to take proven, research-based programs in math and science education and 
successfully demonstrate that they can be replicated on a national scale, state by state, university 
by university. NMSI has raised private funds to initiate replication of two proven programs and 
is currently supporting programs in 14 states. 

The model has the following essential elements: 


Did you know 
there is a 
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+ Selecting programs already proven to be successful with documented results 

+ Identifying the key components that make the program work, which can then be 
replicated 

+ Settling the question of cost and cost-effectiveness 

+ Selecting the right entity to manage and implement the program 

+ Instituting performance management and rigorous program monitoring 

+ Establishing simple and effective collection of evaluation data 

+ Building partnerships and advocacy 

+ Communicating success 

Selecting Proven Programs 

A critical first step is to select programs that are good candidates for replication. Funding 
pilots is an important endeavor, producing innovative ideas and creating knowledge. Pilot 
programs must precede the Scaling Effective Programs effort. They are necessary to determine 
program effectiveness through measurable results. 

What do you look for in selecting a program to replicate? There are several key characteris- 
tics necessary for successful replication, as follows: 

+ At least five years of measurable success. Many programs can produce initial results, 
but do not lead to continued improvement. Furthermore, far too many programs have 
inadequate evaluation designs. The gold standard is the randomized control evaluation. 
Very few programs meet this rigorous standard because implementing programs in 
schools normally does not lend itself to the use of randomized control groups. 

+ Evaluation results based on objective measures such as stan- 
dardized test results, increased graduation rates, increased 
student acceptance into higher education etc. They should 
not be based on satisfaction surveys, anecdotal observations, 
or other “soft” evidence. 

+ Evaluation results that are easily understood and show 
“significant” and “sustained” improvement. Results with 
“statistical significance” often pass muster in technical evalua- 
tions, but may often be too small to pass the “practical signifi- 
cance” test. Practical significance means that the results are so 
pronounced that the average person can easily grasp the impact. The chart that follows 
shows significant increases in the number of Advanced Placement passing exams pro- 
duced by the Advanced Placement Training and Incentive Program in Texas. Notice that 
the improvement not only is marked, but it also continues to increase each subsequent 
year. Such results clearly pass the practical significance test. 


"Cool" pilot 
programs 
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Advanced Placement Passing Scores Continue to Improve 
at the Original 10 Dallas Schools. 



Identifying the Essential Elements of Success 

There are many reasons why a program may produce successful results. Some are simple 
to identify, such as having a dynamic leader who motivates people to perform at very high rates. 
Some may result from a convergence of events - support from a citizens’ education reform 
movement, a sweeping legislative initiative - that produce the right environment for change. 

For a program to be taken to scale and then sustained, there must be a common core of 
factors that can be pinpointed as the indispensable levers that produce the significant results. 
Without having this list of essential elements, a program is not a good candidate for Scaling 
Effective Programs. 

Once identified, these success factors provide a core pro- 
gram that must be preserved in any replication plan. There may 
be allowable variations that make a successful program better 
suited for different environments, but the essential elements 
must be maintained - or you lose valuable benchmarks to deter- 
mine faithful replication. 

The indispensable core components of a program can be 
identified by experts who have experience analyzing and evalu- 
ating performance. An important part of the program selection 
process is to ensure the program staff or advocates can explain 
the essential elements leading to success and articulate why they 
are truly essential by relating them to the performance data. 

Identifying the core factors of success is tantamount to describ- 


How do you 
determine if cer- 
tain components 
of a program are 
necessary to its 
overall success? 


3 



ing the architectural framework of the program to be implemented. Putting those features in 
place also will provide the “milestones” to measure for successful replications. Marking those 
milestones provides a critical path for new adopters to follow, along with the partners and gate- 
keepers who will be needed to sustain the program. 

Settling the Question of Cost and Cost-Effectiveness 

Some programs are simply too expensive to be good candidates for replication and scaling. 
Whether or not the cost is acceptable is a value judgment. Such a judgment relates to the demon- 
strated impact of the program and to the priorities of the eventual public or private funding 
entity. The funding structure is also important. For example, if a program is based on a continu- 
ing public/private partnership at the local level, the tolerance for cost may be higher. If the 
program requires mostly public dollars, those costs must be considered in light of competing 
education- related costs. 

There are at least five ways to look at the cost of implementing programs. These are: 

+ Total lump sum one-time investment 

+ Total recurring investment 

+ Cost in comparison to other public programs that 
provide basic services 

+ Per participant or student cost 

+ Considering the value added from the program com- 
pared to the value added by other programs 

All five must be analyzed when the time comes to scale 
up a program in the public arena. Costs may be covered in 
different ways: such as redistributing resources, generating a 
dedicated funding source, and requiring matching funds for 
public entities who want to implement the programs. Start-up 
costs are almost always greater than continuation costs. Each 
should be examined. 

One rule of thumb is to determine the percent of the total per student cost of any candi- 
date program. If the per student cost is less than 10 percent of the total per student cost in the 
public education system, then the program is within reach, especially if the program is designed 
to reach a certain student population rather than all students. 

Per student funding analysis is important if a school district or state is to eventually adopt 
the program. Such adoption must not be too expensive for a school district to sustain. Any pro- 
posed increases in per student funding competes with the general fund, which supports teacher 
raises and other fixed costs. Programs designed for smaller subsets of students can tolerate 
greater per student costs than programs for the entire population. 

Analyzing the value added from a program compared to the value added by other 
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programs is another way to determine cost effectiveness. There are several programs that are 
intended to produce more engineering majors. 

Consider the following two examples: 

+ Program 1 has had 4,000 students, of which 2,000 went on to major in engineering. 

The total annual cost of this program is $4,000,000. 

+ Program 2 has had 2,000 students, of which 500 went on to major in engineering. The 
total annual cost of this program is $100,000. 

When considering the data above, it would appear that program 1 is better, as it has more 
students in the program and a greater percentage of students who go on to become engineers. 
However, program 1 has a cost of $1,000 per student ($2,000 if you just count the ones who go 
on to major in engineering). Program 2’s cost is $50 per student ($200 if you just count the ones 
who go on to major in engineering.) 

Therefore, while program 1 appears to be more productive, it’s not likely to be a candidate 
due to its high cost in comparison to program 2. And program 2 is not likely to be a candidate 
due to it not being as productive as program 1. 

Discussions with the candidate program operators regarding perceived cost effectiveness of 
the program and issues regarding wider adoption (such as economics of scale) will help funders 
work through this step. Failing to exercise due diligence on the current and anticipated costs 
may result in loss of investment in terms of scaling a program and the loss of the program once 
funding is discontinued. 

Selecting the Right Entity to Manage Replication 

Another critical consideration in this process is selecting the 
right entity to fund. To be successful, a grantee should have the 
capacity to: 

+ Manage the program selected for scale-up 

+ Keep program replication of essentials as a priority 

+ Provide expert support needed to successfully replicate 

+ Attract partners and influence policymakers to produce 
scale-up resources 

+ Understand and establish an accountability system for 
meeting benchmarks and timelines 

The National Math and Science Initiative has chosen to fund education not-for-profit 
entities in selected states to replicate an AP Training and Incentive Program that significantly 
increases student enrollment and success in Advanced Placement math and science courses. For 
the UTeach Program, which prepares content majors to become secondary teachers, we have 
funded universities with large numbers of math and science-related majors. 


Strong manage- 
ment is the key 
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Sometimes large government agencies have too many programs, priorities, restrictions 
and rules to be good candidates to manage program scale-up. However, they can certainly be 
indispensible partners in a competitive process. 

It is always best to select the grantee using a competitive grant selection process. In doing 
so, the funder can evaluate fully the critical capacity issues of all applicants. Important compo- 
nents of the grantee selection process include: 

+ Experience in the relevant field 

+ Excellent leadership 

+ Program management capacity 

+ Commitment to the mission 

+ Ability to expand to serve new sites in subsequent years 

+ Ability to convene influential government and private parties. 

An entity responsible for replication also should have the ability to contract for expert 
assistance in the essential elements of success. These elements may include professional develop- 
ment, setting high standards, observing and providing feedback to teachers, identifying and 
providing interventions to correct problems, etc. An entity should be judged by matching 
its capacity with the need for successful scale-up. Many programs have failed due to lack of 
ongoing support and quality control during initial implementation. 

The RFP process can be designed to determine the capacity of competing applicants. 
Entities that lack any of the required resources or capabilities increase the risk of failure. 
Candidates with no track record or entities that are totally dependent on the grant are risky 
partners. Regardless of the grantee chosen, the funder should lay out the plan of action, funding 
allocations, performance standards, and all other expectations. Success requires a partnership 
between funder and grantee that is built on good communication, trust, and joint problem 
solving. The funder has an intense interest in success that should be shared by the grantee 
partner. 

The Scaling Effective Programs Business Model 

For a program to be taken to scale, there must be demon- 
stration of local support for implementation. Projects totally 
funded by one source often disappear once the funding period 
has expired. Requiring some level of matching funds is a good 
way to secure this local commitment. There is no rule as to the 
level of match most appropriate. However, anything under 25 
percent may be too small of a commitment. 

In addition to a match, funding should be based on per- 
formance. Setting clear performance benchmarks is an incredibly 
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useful management tool for grantees. If periodic release of funds is tied to benchmarks, then 
their attainment becomes priority one. These benchmarks should be those demonstrating 
implementation of the essential program elements. Some benchmarks, if missed, may jeopardize 
the success of a program. For example, one benchmark for the NMSI AP Training and Incentive 
Program is to have students register for fall AP courses in the spring. If this benchmark is 
missed, there will be no program expansion in the fall and a full year will be lost. Managing 
critical benchmarks like this one is the key to keeping implementation on track. 

Programs should be allowed to start on a smaller scale, expanding as they become more 
proficient. A funder’s financial commitment to the scale-up program will determine the level and 
timing of increased funding as well as the gradual withdrawal of funding as the program begins 
to go to scale. NMSI uses a model of three years of gradually increased funding followed by a 
three-year phase-down as the replication site moves to self-sufficiency with local funds and other 
resources. The model used by NMSI calls for a 75, 50, and 25 percent level of funding over a 
three-year period. 

Gradually increasing funding allows grantees to get on solid footing with a manageable set 
of sites before expanding to other sites. It also allows them time to collect performance data, 
which is critical to advocating scale up. The phase-out period gives grantees time to add donors, 
as well as acquire public funding to institutionalize the program. Chances for state or school 
district adoption are heightened if the required investment is smaller and grow as more success 
is demonstrated. 

Monitoring for Success 

Regular program monitoring is not unique to this type of 
funding model. It is a necessary part of any investment. Funders 
should receive regular progress reports and conduct site visits and 
financial audits to ensure fidelity. If a funder does not have the 
in-house expertise to directly monitor program implementation, 
a well-selected grantee or third party may fulfill this role. To the 
degree possible, all progress reports should contain objective data 
linked to the completion of performance benchmarks. 

Some forms of program monitoring can become overly 
burdensome on the grantee, taking attention away from implemen- 
tation activities. Using a negotiated monitoring plan based on 
efficiency and common sense is the best approach. It is critical to identify early warning signals 
on those activities that the success of the program depends on. 

Strategies for Scaling a Program 

If a funder strongly believes in a program, why not set a goal of widespread adoption? 
Accomplishing large-scale adoption includes some variables that are beyond control. Timing for 
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reform is critical - local constituencies must be ready for change. 

The state of the economy is critical. The political goals of gover- 
nors and other elected officials are important. All of these have 
relevance and must be taken into consideration in identifying the 
right time for advocacy. 

A strong communication plan that demonstrates success 
with hard data and personal stories is an absolute must. The devel- 
opment of such a plan should be included in the benchmarks and 
should have resources dedicated to it. 

Advocacy, as used here, means ensuring that those in deci- 
sion-making positions hear about the program, see it in action, 
receive periodic updates, receive performance reports and even serve in partnership roles. 
Advocacy means taking the time to work on scenarios demonstrating the benefits of large-scale 
adoption. 

There is no substitute for strong, influential partners. Partners who serve as grantee 
board members, as donors, mentors, or in other capacities, can be effective advocates within 
their circles. Having advocates who are not paid by the program and who may be personally 
contributing to the program carry greater weight than program staff. 

Program Evaluation 

Taking a program to scale is impossible without reliable clear evaluation data. Most funders 
require some form of evaluation. Evaluation standards should be similar to those used in select- 
ing any program to fund. The goal of the program evaluation here is to demonstrate successful 
replication for potential scale up. 

A complex evaluation system with too many variables, or too complex of a data analysis, 
may place an unnecessary burden on the grantee and, more importantly, the implementation 
sites. The primary focus of all involved must be on successful replication. 

At the same time, it is essential to have processes in place to collect reasonable objective 
data on performance. The goals of the program should determine the primary evaluation 
questions and the evaluation system should be tied to the essential elements of success. During 
the early stages of implementation, outcome data will not be available. However, measuring 
the attainment of benchmarks will be the basis of ongoing progress reports and should include 
objective indices as much as possible. For example, in the case of the AP program, the reporting 
of the completion of student registration for AP courses would include the number of students, 
demographics and types of courses disaggregated by school will provide a robust progress report 
for donors. All results should be objective, easy to understand and communicated through charts 
and graphs with plain language text. Reportable program results are essential for producing a 
sustained program. 
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Summary 

Planning for an investment to have lasting impact takes time and resources. But doing so is 
a good way to protect an investment from going the way of too many projects. Philanthropists, 
foundations and corporate donors have an intense desire to make the world a better place by 
sharing their hard-earned resources. The public sector and many people who struggle to make 
their way in life depend on this generosity. That’s why investing in programs with proven ability 
to make things better and the capacity to be replicated to reach more people, is a worthy goal. 

The National Math and Science Initiative is proving these principles right now by banking 
on proven programs - within one year, NMSI has expanded AP Training and Incentive Programs 
to 67 schools around the country. This has made possible the enrollment of 13,000 more 
American students in more rigorous AP math and science classes, which will help them succeed 
in college and in their careers. At the same time, NMSI has 
expanded the UTeach program for training math and science 
teachers to 13 American universities. That means that 500 more 
students will pursue math and science teaching careers in fall 
2008, providing a desperately needed infusion of teachers with 
content knowledge and certification. 

This is tangible, significant progress. It is made possible 
by scaling effective programs and taking the best education 
approaches to more American students. With greater support 
from the public and private sector, it is a model that could trigger 
positive results around the country. 


At the end of the 
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